@yannis-roussos I'm going to revert this change -- it was syntactically/idiomatically correct as it was. You downsample, and the benefit you get "in exchange for downsampling" is increased storage space. It's a trade-off.

I think the "in exchange for" structure is idiomatic and concise, but we could spell it out more if you prefer:

"Downsampling lets you reduce the resolution and precision of older data, in order to free up storage space."
or
"Downsampling lets you reduce the resolution and precision of older data, in order to also reduce the data's storage footprint."

The last option seems good. WDYT?

@marciw OK, I get what you are saying! I agree, the way this is written can confuse people in the same way I got confused. Both proposals look good to me, whichever on you prefer looks good to me!

@gmarouli is this true for ILM? If my understanding is correct, this is only true for downsample actions on the hot phase (coupled with rollover) or for ones with migrate not enabled.

If I am not wrong, can you help with rephrasing this sentence?

Yes, this is true for ILM.

@yannis-roussos the downsampling action is executed before the migrate action so it happens on the same tier as the original index.

This has caused confusion in the past, but the benefits outweigh the cons:

• The "hotter" the tier the better the resources it has, the more performant the downsampling would be.
• Migrate will have less data to "migrate".

Thank you Mary, your explanation helps a lot!

Should we add part of it here so that we don't confuse our users?

@yannis-roussos sorry, what exactly do you want to add here?

I am curious to hear what @yannis-roussos has in mind, from me this looks clear enough.

I was thinking something more or less like what you explained to me and it immediately made sense:

Not perfect, but I hope that it provides some clarity on the way I was thinking about this. We can of course skip this suggestion if you think that it is overly descriptive. But if this caused confusion in the past, maybe it is worth noting somewhere

I am concerned that most users do not know about the migrate action, this why just saying that it runs at the tier of the original index feels more helpful to me.

I have some concerns about the second sentence too. It feels like we are defending an implementation detail to our users, trying to say "it might feel weird but it has its benefits". I feel the docs is not the right place for this, if someone has concerns we can always redirect them to the relevant github issue. If we put it in the docs it feels like we are asking for reactions. For example, we do not explain to the users why the downsampling requires the data to be read-only, we just say it does. I think this is similar.

Makes sense Mary, let's keep it as is then!

I propose to keep the same approach as in the quickstart guide and provide real examples instead of a generic meta discussion on something I have not yet seen as a user

• You can query multiple raw data and downsampled indices in a single request, and a single request can include downsampled indices with multiple downsampling intervals (for example, 15m, 1h, 1d).
• When you run queries in {{kib}} and through Elastic solutions, a standard response is returned, with no indication that some of the queried indices are downsampled.

This more or less tells the user that (1) data streams can have multiple downsampling granularities set for them and that (2) you can just query a data stream as usual.

I think that we should:

• introduce this notion of multiple granularities in different phases with an example first somewhere on a section before this one. (Maybe an additional reason to send this section after a page for setting up / running downsampling?)
  and then
• just tell the user that they should not worry about downsampling. Querying over multiple downsampling granularities just works and, in general, you will not even get an indication that it is there

@gmarouli could you help with a few ideas on what we could have in this section?

I agree that this is going a bit deeper than the rest of the document. Can we convey our message with an image? Or list the things that cannot be done with downsampled data?

I am not opposed to examples, but they need to be carefully chosen to be represent the exact cases we want, for example the timezone offset.

@yannis-roussos @gmarouli Can one of you try to draft the changes you're suggesting or propose some concrete examples? I did create the separate page, so you'll see that the next time you review.

The multiple granularities were addressed. About the querying, the text is also simpler now, so I am not sure, if we want to continue with examples or not. @yannis-roussos what do you think?

I agree, the new dedicated page for querying downsampled data looks much simple now. Maybe just add a note to link to the TS command once it is ready (as an update included in elastic/elasticsearch#134373)?

This is only true for running downsampling manually ("Downsampling with the API" tab). ILM and DLM will take care of this on behalf of the user.

Am I right @gmarouli?

In general I am not sure if we want to get into the manual downsampling that much outside of the dedicated example (and even that one feels like to much to me on the old docs).

I propose to be opinionated and just stick to the happy path / what we propose: Today it is ILM (I guess) and we can change that to DLM once we have tiers supported.

I propose this to move under the "Downsampling with the API" tab

I would also add a little bit more info on what are the upsides / downsides of each method - everything that we have been talking about on the incremental downsampling docs @gmarouli but are not included anywhere in the canonical docs for the current downsampling. Like that we can only have as many actions as phases in ILM or the fact that we have a higher limit on DLM but no ability to move between tiers.

What do you think @gmarouli?

I agree with @yannis-roussos in most comments. But it's not so simple either.

To start with, I think we need to agree about the scope of this page. Is it to guide the user on how to downsample their time series data streams or how to downsample a single index. These two things are not equivalent.

• Downsampling a single read-only index means just execute the downsampling API and you get a new index, now you have two indices the original and the downsampled one. But effectively you have downsampled the index, what you want to do with that is up to the user. I would ONLY recommend this for experimentation, if a user wants to see how downsampled data look like, I would suggest that they do that and they play with the downsampled index.
• On the other hand, downsampling a data stream with ILM or data stream lifecycle, has many other steps such as waiting for the end time of a backing index to pass, making the index read-only, executing the downsampling, replacing the original backing index with the new one and then deleting the original one. Yes, the downsampling operation is executed on the backing indices but there are other actions that happen on the data stream.

My recommendation/preference is:

• The scope of this doc is downsampling time series data streams.
• We should mention that downsampling requires the data to be read-only, so when a time range is downsampled it cannot process any more updates.
• We only talk about data stream lifecycle and ILM. We can start with a sentence to help them choose which one to look at, for example, "serverless only look at data stream lifecycle, if you need tiers in stack look at ILM. If you do not need tiers consider using data stream lifecycle because it supports more downsampling rounds".
• We start with data stream lifecycle and we show only the lifecycle configuration not the whole composable index template, this feels more comparable to the ILM policy.
• We finally go through ILM.

What do you think?

I agree with Mary's proposal and I ++ the fact that this is about downsampling time series data streams. Downsampling a stand alone index is an advanced operation that is not part of the path that we want to guide users towards.

I tried to address some of this, but please take another look -- not sure I'm following your comments.

I am curious to hear what @yannis-roussos has in mind, from me this looks clear enough.

@marciw do you think we should mention here also that the index time series end time will be respected?

Should we mention here that a user can add it to the index template but it will work for new data streams only?